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Abstract 



This note gives a short review of the statistical issues concerning upper limit calculation 
O ■ and claiming of discovery arising in the search for exotic physics with neutrino telescopes. 

Low sample sizes and significant instrumental uncertainties require special consideration. 
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Methods for treating instrumental or theoretical uncertainties in the calculation of limits 
■ or discovery are described. Software implementing these methods is presented. The issue 

of optimization of analysis cuts and definition of sensitivity is briefly discussed. 



1. Introduction 
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Searches for exotic physics in neutrino telescopes pose challenges to statistical 
inference. In neutrino telescopes often physicists have to deal with low event counts 
and significant instrumental uncertainties which need to be taken into account. 
\ The small sample size makes the use of asymptotic methods doubtful. Presence of 

significant instrumental or theoretical uncertainties introduce nuisance parameters 
which need to be considered in the estimation of physics parameter intervals or in 
claims for discovery. 

Neutrino astronomy has so far been in an infant state, the main aim of existing ex- 
(— > ■ periments being proof of principle with little hope of detecting neutrinos of cosmic 

and/or exotic origin. Analyses have therefore often been optimized for setting the 
most stringent upper limit. With the advent of the IceCube detector this situation 
might change, in the sense that discovery should become more likely. Analysis op- 
■ timization methods should therefore be reviewed. 

In the next section, we remind the reader of methods to calculate confidence in- 
tervals and to claim discovery. In section 3 we present two methods for including 
uncertainties, followed by a discussion on analysis optimization. 
For simplicity, unless otherwise stated, throughout the paper we will consider a sta- 
tistical process with probability density function (PDF) P(n\s), i.e. the probability 
to observe n given the parameter s. 

2. Claiming discovery and calculating confidence intervals 

Though often mixed up [1,2], claiming discovery and calculation of confidence 
intervals are generally presented as two different cases in statistical literature, see 
e.g. [3] for an exhaustive introduction. 

Claiming discovery Claiming discovery is an example of hypothesis testing. In the 
search of exotic physics the null hypothesis, Hq, is usually "No signal present" and 
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the alternative hypothesis, Hi, is "Signal of exotic physics present". Discovery is 
claimed if the measurement result would be very unlikely under the condition that 
the null hypothesis is true. This is quantified by calculating the p-value: 

P = P(T>T obs \H ) (1) 

where T denotes a test statistics, which is a function of the observation and the 
involved hypotheses (see below) , and P is a probability density function describing 
the distribution of the test statistics under the null hypothesis. If the p-value is 
smaller than S01116 predetermined, threshold, ctsigm 

then the null hypothesis is re- 
jected. A particular common choice of the threshold is the probability corresponding 
to 5<7 in an equivalent Gaussian distribution, i.e. ct s i gn ~ 3 • 10~ 7 . 
The test statistics which in a statistical sense is most optimal in distinguishing two 
alternative hypotheses is according to the Neyman-Pearson lemma the likelihood 
ratio: 

C(n\H ) 

C{n\Hi) K ' 

One useful property of the likelihood ratio is that under the assumption that the 
null hypothesis is true, asymptotically the following statement holds: 

-21nT~ X 2 (3) 

i.e. minus twice the logarithm of the likelihood ratio follows a \ 2 distribution. 
Though this property is often used in particle physics it is not always true. Except 
for normality, there are a series of other requirements, some of which are violated 
even for quite common cases [2] [4] . An experimenter intending to use the \ 2 dis- 
tribution should therefore be convinced that this is justified and in case of doubt, 
calculate the distribution of the test statistics under the null hypothesis for example 
by using Monte Carlo simulations. 

The threshold probability in order to decide if the null hypothesis should be re- 
jected is based on the number of false detection one is willing to accept and on the 
number of searches which are performed. For example, in the context of one of the 
large CERN experiments, there might be about 50000 independent channels [5], i.e. 
500 channels, 1000 resolution elements each. This implies a false positive detection 
rate of about 1.5 % if one requires a 5er detection. It is not obvious, that a similar 
reasoning would lead to the same requirement in case of searches with neutrino 
telescopes. 



Calculation of confidence intervals Consider s a fixed (unknown) parameter. In 
frequentist statistic^, one defines a confidence interval [si,S2] as a member of a 
set of intervals for which: 

P{s 6 [si, s 2 ]) = 1 — oe , for all s (4) 

These sets are to be constructed using only P(n\s), s being a fixed but unknown 
parameter and not P(s\n). The latter would make the method Bayesian, since s is 
treated as a random variable. The frequentist construction has been introduced by 
Neyman [6] . A special case, which is very common in high energy physics has been 
proposed by Feldman & Cousins [7]. Other common approximative methods exist 

1 the Bayesian calculation is somewhat simpler, since here the interval can be found by integrating 
in the probability density function P(s\n), where the parameter is considered a random variable 
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which make use only of the likelihood function, e.g. [3,8]. 

All frequentist methods which fulfill the condition in equation 4 are said to have 
coveraa^i Methods which do not have coverage are not valid or need correctional 
In case of doubt, coverage has to be demonstrated for example by using Monte 
Carlo simulations. 

3. Treatment of nuisance parameters 

Nuisance parameters are parameters which enter the data model, but which are 
not of prime interest. The probably most common example is the expected back- 
ground in a Poisson process. In a usual physics experiment, only confidence intervals 
on the parameter of primary interest (for example signal flux or cross-section) are 
of interest. Thus, ways have to be found to marginalize the nuisance parameter. 
There are two common approaches: 

In the first method, the PDF without uncertainty in nuisance parameters is replaced 
by one where there is an integration over all possible true values of the nuisance 
parameter (integration method): 

/>oo 

P(n\s,b t rue) ► / P{n\s,btrue)P{btrue\best)db tr ue (5) 

JO 

Here btme is the true value of the nuisance parameter and b est is its estimate. Since 
the integrated PDF is describing the probability of the true value given its estimate 
(and not vice versa) this method is Bayesian. Some prior probability distribution 
of the true value of the nuisance parameter has to be assumed. 
In the other common method, the PDF is replaced by one where for each s the 
PDF is maximized with respect to the nuisance parameters (profiling method) 

P(n\s,b t rue) — ► m&x£(n\s,b true ) (6) 

with notation as above. This method is completely frequentist, since it never treats 
btme as a random variable. Therefore the argument of the maximisation is a likeli- 
hood function and not a PDF. 

Both approaches have recently been subjected to detailed studies regarding their 
coverage, i.e. with applications to confidence intervals [9,10,11]. For typical problems 
arising in neutrino telescopes they perform satisfactory. In the context of the LHC 
searches there are indications the Bayesian method under-covers badly, whereas the 
profiling method still seems to work fine [1]. 

Software has been developed which implements the integration and profiling method 
for typical problems arising in neutrino telescope analyses. 

pole++ is a C++ library of classes which was developed based on the method 
presented in [12]. It allows calculating confidence intervals using the Feldman & 
Cousins method with integration of nuisance parameters and coverage studies. The 
signal process considered is a Poisson with known (possibly uncertain) background 

2 To be more precise, in frequentist statistics in addition the probability has to be defined in 
terms of repeated identical experiments. 

3 In general, physicists tend to accept over-coverage (meaning, being conservative), whereas under- 
coverage is considered unacceptable. 
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with different models for nuisance parameters. Several experiments with correlated 
or uncorrelated uncertainties in the nuisance parameters can be combined. The 
pole++ library can be obtained from http://cern.ch/tegen/statistics.html 
TRolke is class which is part of the ROOT analysis package [13]. It treats a Poisson 
signal with background process with seven different models for nuisance parameters, 
which are marginalized using the profiling method. For extraction of the confidence 
intervals the likelihood function is used. For a description of the method see [10]. 
It should be noted that the MINUIT package [14] (in particular using the MINOS 
facility) also applies the profiling method and is completely general as long as the 
likelihood function can be written down. For the special case of a Poisson process 
TRolke applies some improvements. 

Though profiling and integration are here presented in the context of confidence 
intervals, both approaches can also be used in hypothesis testing. The substitutions 
in equations 5 and 6 then have to be applied to the likelihoods. The statistical 
properties to be studied in this case would be the distribution of the test statistics 
under the null hypothesis as well as possibly the power function (see next section). 

4. Analysis optimization 

Analyses are optimized defining some figure of merit (FOM), which will be max- 
imized (or minimized) with respect to some cut value t 

In searches with neutrino telescopes, it is often chosen to try to set the most strin- 
gent upper limit leading to the introduction of Model Rejection Factor [15]: 

MRF = < Sl - a > (7) 
n s 

Here, si_ Q denotes the (1-a) confidence level upper limit on s. The mean is taken 
over the Poisson distribution with no signal. In case of presence of uncertainties in 
nuisance parameters the corresponding mean becomes: 

POO POO 

< si-a > = / / dndb es tSi- a P(n\b es t)P(b es t\b tr ue) (8) 
Jo Jo 

Unless the size of the uncertainties depends on the cut value or assymetric PDFs 
for the nuisance parameters have to be considered, the mean calculated using only 
the Poisson distribution should be sufficient for the calculation of the FOM. 
In case a decision on whether to set a limit or claim discovery beforehand is not 
desired, the figure of merit will have to be based on a sensitivity region which is 
meaningful both if the experimenter would report a limit and if the experimenter 
wants to claim discovery. Punzi [16] suggests to define the sensitivity region by: 

1 -&..„„(*) > 1-acL (9) 

Where a S i gn is the significance which the experimenter requires to claim discovery 
and etc l is the confidence level required in case the experimenter wants to calculate 
a limit. j3{s) is the so called power function. Power is a concept arising in hypothesis 
testing. The power of a test is the probability to reject the null-hypothesis given 
the alternative hypothesis is true. The concept can also be applied to confidence 
intervals [11]. This definition of sensitivity fulfills several desirable properties: for 
example if s is inside the region defined by equation 9, then there is a probability 
of at least 1 — oicl that it will be discovered. From the sensitivity a FOM can be 
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calculated. Simple expressions of the FOM for common problems can be found in 
[16]. 

5. Conclusions 

Major statistical challenges to be faced in searches for new physics with neutrino 
telescopes are low statistics, large instrumental uncertainties and the desire to de- 
tect an unknown process while at the same time being able to set stringent upper 
limits. 

In this note we point out and briefly discuss different solutions to the above chal- 
lenges. There are several methods existing to calculate confidence intervals in the 
presence of instrumental uncertainties which have been tested and behave well also 
for the small sample sizes and relatively large uncertainties usually encountered in 
neutrino astronomy. Code performing the necessary calculations is readily available. 
Sensitivity regions linking the power of a hypothesis test with the confidence level 
of a confidence interval yield figure of merits which can be used to optimize analysis 
with respect to both discovery and stringent limits. 
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